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Due to rapid information and communication technology (ICT) growth, 
government agencies must manage more digital apps to support public 
service operations and administration. In Indonesia, at least 400,000 
applications have been received by various ministries and government 
agencies. This amount will hurt the ICT budget’s investment and waste if no 
approach is employed. Problems can be solved with application portfolio 
management (APM). Indonesian government agencies’ AMP 
implementation is unclear. APM is applied at the government research 
institute agency for the assessment and application of technology (BPPT) in 
this study. BPPT was chosen due to APM’s lack of ICT investment 
management. This research examined 41 submissions from the 2019 digital 


management transformation initiative. APM selected two mapping models. The outcome 
Gartner’s TIME model indicates how APM may offer ICT strategies for current applications to 
IT planning government entities. This analysis mapped existing applications into two 


models: McFarlan’s strategic grid and gartner’s TIME model. Mapping 
findings from these two models-technical health evaluations and regulatory 
compliance-may be used for application sustainability suggestions. 11 
treatments were advised for maintenance and investment, while 4 
applications were recommended for removal. This research helps us 
understand how the Indonesian government institute maintains its 
application portfolio and how APM might be a valuable method for 
application management. 
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1. INTRODUCTION 

In July 2022, Indonesia’s Minister of Finance highlighted an enormous number of applications 
owned by Indonesia’s government. A total of 400,000 applications were registered. This staggering number 
was believed to have become a burden on national expenditures. The increasing number of digital services 
and rapid transformation have made it urgent to manage application portfolios and implement strategies to 
ensure that information and communication technology (ICT) investments are practical and efficient [1]. 
Investment decision-making is usually based on several aspects, including planning, redundancy, risk, and 
resources [2]. If not done correctly, issues such as poor ICT investment, redundant applications within 
institutions [3], and bad ICT-related decisions can occur. Application portfolio management (APM) is a 
strategy that can be implemented to reduce these issues. APM is a strategy that an organization can use to 
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classify existing or planned applications that it owns into groups to align applications with the business core. 
APM can further coordinate ICT investment planning [4], lower ICT costs [5], manage risk [6], and manage 
ICT resources so that organizations can align business processes and ICT investment [7]. Recently, the 
practice of APM has grown rapidly to help reduce costs for applications that do not need it [8]. 

Many frameworks are commonly used to implement APM. We focus on the two most used 
frameworks, McFarlan’s strategic grid and gartner’s TIME model. As a national research institution, the 
agency for assessing and applying technology's (BPPT) duty is to enact government roles in research. To do 
so, BPPT is supported by various applications spread across departments. According to our independent 
observation, BPPT has not developed an application portfolio management strategy for existing and planned 
applications. With the digital transformation program adding 41 new applications to support management, 
BPPT is vulnerable to poor application management strategy issues. 

Surprisingly, there is little information available about how Indonesian governments manage their 
existing applications. As a result, this study aims to fill the gap by revealing how BPPT, Indonesia's research 
institution, manages its application. APM is also applied to the proposed 41 digital applications. We contend 
that BPPT necessitates APM in order to ensure that ICT investments are made effectively and efficiently. 
The following questions will be addressed by this research: 

— How does BPPT implement the APM strategy for the 41 digital applications? 

— How are the 41 digital applications mapped using McFarlan’s strategic grid? 

— How are the 41 digital applications mapped using GARTNER’s TIME Model? 

— What kind of recommendations towards the 41 applications based on the two frameworks? 

The following sections are a literature review in which a discussion on APM methodology will be 
introduced along with the two frameworks that will form the basis of our research. Still, in the same section, 
a brief introduction to BPPT and its 41 digital applications will be discussed. Section three is the research 
methodology section, followed by the results and discussion sections. Our last section is the conclusion. 


2. LITERATURE REVIEW 
2.1. Application portfolio management 

Portfolio management is a step related to a product, including the evaluation process and priority 
selection to determine the priority of a product so that it can reallocate available resources [9]. Portfolio 
management is often used to make appropriate, effective, and efficient decisions. This method has been 
widely used in various sectors, such as water supply [10], project management [11], tourism management 
[12], and information technology [13]. 

Application portfolio management (APM) is part of information technology portfolio management 
that focuses on software that directly supports business processes. APM is a process carried out by 
stakeholders as the basis for making decisions regarding existing organizational applications [3]. Previous 
research [14] stated that APM successfully simplifies the process of creating application portfolios to manage 
complex application portfolios. The first stage of APM is carried out by collecting a list of all applications 
that have been developed or are still in the development process. The data owner and users of each 
application are then identified. Furthermore, the application is assessed based on its level of use by 
rationalizing it as a whole. 

The list of collected applications is then carried out in an inventory to determine each application’s 
total cost and business value. At this stage, business capabilities determine what the business is doing today 
and what needs to be done to meet current and future challenges. Based on the results of the previous stages, 
an application architecture framework is created by defining a set of business, information, and application 
concepts that the organization wants to see in the long term. These results are then re-aligned with the 
information technology (IT) roadmap at the institution. 


2.2. McFarlan’s strategic grid 

McFarlan’s strategic grid is a strategy for mapping applications into a grid based on their 
contribution to the organization [15]. In a previous study [16], [17], a grid was divided into four categories: 
support, key operational, high potential, and strategic, as shown in Figure 1. Each category describes each 
application’s contribution to the organization. Bintang et al. [18] mapped all the applications to McFarlan’s 
strategic grid based on identification by the general good of the infrastructure, human resources, and portfolio 
of applications. Research conducted [19] uses McFarlan’s strategic grid to create a future application 
portfolio for nonfranchise pharmacy companies. An analysis of the functionality of each application to the 
organization is carried out to map each application. 
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Figure 1. Four categories of McFarlan’s strategic grid 


Another study [20] used McFarlan’s strategic grid to analyze the alignment between business 
strategy and information system planning at the API Polytechnic Yogyakarta. The support category is for 
applications that support the ease of business processes but do not affect the organization’s business success. 
Usually, those applications are used for administration and the back office. The key operational category is 
for applications that manage business processes and provide decision support but do not directly affect 
corporate profits. The high potential category is for applications that have potential and value in the future 
but do not currently have a significant impact on the organization. And the last is the strategic category for 
applications that facilitate business processes and significantly affect the organization’s continuity. 


2.3. Gartner’s TIME model 

Like Mc’Farlan’s strategic grid, the gartner TIME model also maps applications into a four-quadrant 
grid. Gartner’s TIME model is used to evaluate ICT investments in a company. T (tolerate), I (Invest), M 
(Migrate), and E (Eliminate) represent recommendations for an ICT application/investment. TIME has been 
known for its ease of application and understanding in executing the APM process. 

The mapping application of gartner’s TIME Model method is based on its business value and 
technical quality [21]. Business value is obtained from the user’s functionality, efficiency, reliability, and 
frequency of use assessment, while IT quality is obtained from compatibility, security, maintenance, and 
portability. After each application gets these two values, a mapping is carried out using 2-dimensional 
coordinates with business values as the x-axis and IT quality (technical efficiency and safety) as the y-axis, as 
shown in Figure 2. 


High 


Low 


bo Oe | High 


Figure 2. Two-dimensional coordinates of gartner’s TIME model [21] 


The first quadrant is "tolerate," in which applications with low support for business value but a 
working function and high IT quality will be put. The Invest category is for applications with high business 
value and IT quality. Applications fall into this quadrant relatively easily, experiencing minor disruptions, 
serving their purpose, providing source code availability, and generating cost savings or income. 

The third quadrant is "eliminate," in which applications with low IT quality and less support for 
business value will reside. The last quadrant is "migrate," a category for applications with high business 
value but low IT quality. The business is using the application and can articulate its value. Still, there is a 
high cost of support, a lack of knowledge, a lack of source code, or a variety of other factors that make it an 
expensive application to maintain. 
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2.4. BPPT digital transformation 

The agency for the assessment and application of technology (BPPT) is a non-ministerial 
government institution charged with assessing and applying technology on behalf of the government. BPPT 
is assisted in carrying out its management duties and functions by a number of applications distributed across 
several units. In order to implement e-government, digital transformation is required. The goal of digital 
transformation is to improve public services and governance while also preparing the government for 
Industry 4.0 and Society 5.0 [22]. In 2019, BPPT began digital transformation activities, one of which was 
the development of 41 management support applications in response to the implementation of Presidential 
Regulation Number 95 of 2018 of the Republic of Indonesia, concerning the Electronic-Based Government 
System. 


3. METHOD 

This study used a mixed methodology to assess the existing 41 digital applications under BPPT’s 
Digital Transformation program. This study used the McFarlan strategic grid and gartner’s TIME Model as 
an analytical framework for application portfolio management. A survey targeting users and developers of 
each application was done in August 2021. Experts with a more profound knowledge of 41 applications were 
invited for an interview. The list of questions was adopted from ISO 25010. 


3.1. Data collection 

Data collection regarding the application is required to find out the current state of the application. 
Information related to the application is necessary to make the application portfolio, so we strive to balance 
the value of the application and the cost caused. The required information includes the application’s name, 
the version used, the function, the number of users, the operating system, the application owner, system 
security, and so on. 

APM data can be collected in three ways: automatically, semi-automatically, and manually [23]. 
Manual data collection can be accomplished through a survey-collection through a survey on several units 
related to the work in charge of the application. The data collection was done in two phases. The data 
collection phase was conducted from July 2021 to November 2021. In the first phase, we collected technical 
information on applications to build an application catalog. The information required includes the name of 
the application, the version used, the function of the application, the user, the operating system, the owner of 
the application, the security system, and others. 

In the second phase, we collected user and developer data through an online questionnaire. The 
survey was conducted to determine the IT quality and business value, which were then used to map the 
application into the gartner TIME model. In this study, business value measurement was determined by the 
ability to solve a business need and provide operational efficiencies, a high utilization degree, and a positive 
user experience. In contrast, the support aspects chose IT quality measurement, data accuracy, source code 
availability and quality, reliability and security, response time and ease of change, and technology. Those 
aspects were adapted from FPT-intellinet [24], as shown in Table | and Table 2. 


Table 1. Application IT quality aspects 


IT Quality Aspect 
The function of the application is to complete tasks following the organization’s function Solves a Business Need 
The time and resources required to use the application Provides Operational Efficiencies 
Applications are easy to use and easy to learn [25] User Experience 
Probability of the application working correctly in a specific environment and time Data Accuracy 
Frequency (average) usage of the app in one week/month/year Utilization 
The level of user satisfaction after using the application [26] User Experience 


Table 2. Application business value aspects 


Business Value Aspect 
The application can operate and communicate well with other applications Interoperability 
Data can only be accessed by authorized users (Confidentiality), changes to data and Security 


application components are accessed by authorized users (integrity), guarantees that users 
cannot deny the operations they perform (non-repudiation), and all actions on the application 
can be traced (accountability), the ability to identify legitimate users (authenticity) 


The ability to maintain without changing or reducing the original application’s quality. Source code availability and quality 
The ability of applications to operate with other infrastructure without requiring high Ease of Change 

modifications. 

The cost that may incur from maintenance Cost Saving 
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A questionnaire adapted from ISO 25010 was made to get the value of these aspects. ISO standards 
are often used to test application quality, such as in research [27]. The survey used a measurement scale of 0 
to 5, where zero indicates no assessment aspect of the application, and five indicates that the application 
shows a very high assessment aspect. 

Our survey respondents came from different working units, such as the Bureau of Organizational 
Human Resources, the Legal Bureau for Cooperation and Public Relations, the Financial Planning Bureau, 
the Education and Training Center, the Inspectorate, and the General Bureau. All respondents were briefed 
on how to fill out the questionnaire and validated as employees working from designated units. The chosen 
experts for the interview were BPPT employees with at least 20 years of working experience (System 
managers?) in the IT field and had a deep knowledge of the 41 applications. These experts agreed to a 
recorded online interview. 


3.2. Survey and expert result 

There were 102 data collected for 25 applications. All data were manually validated. The developer 
survey collects information to get the value of IT quality. Aspects that are assessed include compatibility, 
security, maintenance, and portability. Previous research stated that interviewing with experts is one of the 
most effective ways to gather information about the application [28]. Information was obtained by combining 
survey data collection and expert interviews to prevent bias [29]. Our experts put weight on each IT quality 
and business value criterion, as shown in Table 3. Furthermore, each of our experts classifies 25 applications 
into four quadrants of McFarlan’s strategic grid. The results will be discussed in section 4.4. 


Table 3. IT quality and business value aspects weight 


IT Quality Weight (%) Business Value Weight (%) 
Support 12.5 Solves a business need 27.5 
Data Accuracy 30 Provides operational efficiencies 25 
Source code availability and quality 17.5 Provides critical function 15 
Reliability/Security 15 Utilization 15 
Response time/Ease of Change 75 User Experience 15 
Technology 17.5 Cost Savings 2.5, 


4. RESULT AND DISCUSSION 

Among the 41 applications used in digital transformation activities at BPPT, 16 were not included in 
the analysis because the developer could not be contacted or the application was not used when data 
collection was carried out. Only 25 applications will be used for analysis. An application survey was carried 
out on 25 existing applications to obtain technical details. 


4.1. Incomplete documentation 

Good software documentation provides important information regarding the description of the 
software, what the program does, and how it should run [30]. In the end, incomplete documents will 
complicate software development and maintenance in the future. In fact, developers often override this issue. 
Throughout the data collection phase, we experienced difficulties collecting complete documentation. We 
asked our application owner to submit at least six documents. These documents are business process, SRS, 
user acceptance test documentation, user manual, system documentation, and user role documentation. Only 
12 out of 27 applications are complete with a business process document. While 10 out of 27 is assisted with 
SRS document, 13 out of 27 have the source code documented, and only 3 applications have database 
documentation. 

This incomplete documentation showed that the software development process at BPPT put aside 
documentation as its slightest concern. Documentation for software development needs to be considered an 
essential factor. In his 11 -page research, [31] argued the importance of user documentation by elaborating on 
a misconception. His study showed that even the most well-designed software needs documentation. In 
alignment with this issue, we found many vendors that were not contactable for completing our survey, and 
most of the submitted documentation was incomplete. This raises the issue of software sustainability because 
the vendor is supposed to supply the application owner with complete documentation. 


4.2. Application catalog 

An application catalog was created using a simple spreadsheet to record the data. Catalogs are 
considered documentation suitable for recapitulating software products simply [32]. The collected 
information was stored in an online storage application with limited access. The catalog consisted of 25 
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applications with detailed information. The application catalog can then be used by the person in charge of 
the application for application monitoring to find resources and simplify maintenance and repair. In addition, 
the purpose of creating an applications catalog is so that development will not depend on employee 
information, which may be lost but depends on documents. An example of an application catalog that has 
been created can be seen in Table 4. 


Table 4. An example of an application catalog 


Attribute Description 

Application Name Sispekin 

Purpose SISPEKIN aims to improve the performance accountability of all entities within the BPPT 
through the presentation of information about the organization, plans and performance targets 
set; performance measurement; evaluation and analysis of performance achievements for each 
predetermined performance plan and target or program/activity results and final conditions that 
should be realized; and integrated performance reporting. 

PIC Rizky Agung W 

Unit Biro RENKEU 

Developer PMI 

Source Access https://sispekin.bppt.go.id 

Develop 2018 

Deployment 2018 

Full Operated 2019 

Active PIC perunit, 4*45= 180 

Admin 5 

Source code type PHP version 7.2 

Server engine Centos 7 

Database engine MariaDB 

Database Name Sispekin/c2dblakip 

Database Address 10.10.180.109 

App Framework Yii2 

Other Script Code Yes 

SSL Yes 

CAS/SSO Yes 

Browser Google Chrome, Mozila Firefox, Microsoft Edge, Opera 

OS - Mobile Y (mobile view) 

OS - Desktop Windows, Linux 

Application placement BPPT - PMI 

Phisical Application placement Serpong 

Application database placement Serpong 


Generated data 
External users 
Business Process 


Quarterly report, annual report, LKIP Report, PJK and PJRO Report 
SP2D, DIPA RKKL Kemenkeu JDH with impor excel manually 
Yes 


SRS Yes 

UAT No 

User Manual Yes 
System Documentation No 

User Role Yes 
ERD/DAD/DFD No 

Data dictionary Yes 
Application Password Available 
Username dan Password DB Available 


4.3. McFarlan’s strategic grid 

The mapping of 41 applications into the McFarlan strategic grid was based on our experts’ 
assessment conducted at the interview and portfolio of applications. The number of applications that fell into 
the support category was 21. It was the most significant number compared to other categories. While the key 
operation category has had seven applications, the strategic category has had 10 applications, and the high 
potential category has had three applications. The result of this mapping, as shown in Table 4 reflects that the 
application function in one of the largest research institutions in Indonesia has not yet played an important 
role. In this case, the applications in BPPT are seen as tools to support the operational implementation of 
research activities. 

The results of this mapping could be considered for setting priorities for ICT investment in BPPT. 
Applications that fall into the “high potential” and “strategic” categoriesy should be a top priority for 
investment. By utilizing the results of this mapping, ICT investment in organizations is expected to be more 
optimal. Many institutions are experiencing increased ICT investments cost and are stuck with poorly 
managed applications. Application portfolio management is carried out to ensure that investments are made 
following the institution’s ICT roadmap. 
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4.4. Gartner’s TIME model mapping 

Due to the limited information, only 16 applications can be analyzed and mapped into gartner’s 
TIME model. Gartner’s TIME Model mapping was done by looking at the results of the IT Quality and 
Business Value assessment results from the survey that has been done previously. As shown in Table 5, 
among 16 applications, only 4 applications fall into the eliminated category, while the rest are categorized as 
invest. From this, we can conclude that digital application in BPPT focuses on the longer-term application 
that can be developed further rather than a one-time-use application. These applications were also highly 
supporting BPPT’s business value. 

As shown in Table 6, four applications were within the eliminated quadrant: e-Public Relations, E- 
KI, E-Arsip, and Kerjasama Online. These four applications show a lack of IT quality and low to average 
support for BPPT’s business value. To date, these applications have been merged into one big application 
with four main functions, as inherited from their predecessors. The next research phase can do further 
analysis to elaborate more on the new application value of the two frameworks. Surprisingly, no applications 
fit the "tolerate" and "migrate" categories. The earlier category is filled with applications that have a high 
quality of ICT but offer little support for the institution’s business value. Our results revealed that BPPT 
focused on developing an application that supports business value. 


Table 5. The result of Mcfarlan’s strategic grid mapping 


Strategic High Potential Key Operational Support 
Monitoring Kepuasan  Pelanggan KM BPPT, Tracking Proses, Permintaan Pandawa, Permintaan  Layanan 
(Customer Satisfaction Monitoring), Sispekin, Layanan Komersialisasi Teknologi (Request of Technology 
Simprotek, e-SMM, Dashboard SINEVA (Commercialization Service Service), Eksis, SIREKA, 
Program dan Anggaran (Program and Request), e-mail dan Berkas SIDUPER, SIPASKAL, SISKA, 
Budgeting Dashboard), Simonbaja, Terintegrasi (e-mail and SIGAGAS, SIEVITA, Siskolapi, e- 
Siskonawas, SPARK, e-Dakin, integrated-file), Fabiola, SIM Salary, SPIRIT, Jdaman, Mersys, e- 
Siskarevo, BMN Plus Cepat Tanggap (SIM Quick Arsip, Sijempol, Kerjasama Online 
Response), e-Sarpras, e- (Online Collaboration), e-KI, e- 
Harmonisasi PPID, e-DISPOSISI, Bels 


Table 6. The result of gartner’s TIME model mapping 
Tolerate Invest Eliminate Migrate 
SIDUPER, Bels, e-Salary, Fabiola, SIDARA, BMN Plus, e-KI, Kerjasama Online (Online 
SIEVITA, SIGAGAS, Sijempol, SIM Cepat Tanggap Collaboration), e-Arsip (e-Archive). 
(Quick Response), SISKA, Sispekin, Spirit. 


4.5. Combine McFarlan strategic grid and gartner’s TIME model 

Our analysis combined McFarlan strategic grid and gartner’s TIME Model to propose a powerful 
recommendation. By far, this research is the first attempt to combine the two frameworks. We first create a 
combination of results from both frameworks. As gartner’s TIME Model only resulted in two categories 
for an existing application, the recommendation resulted in 4 types. The recommendation can be seen in 
Figure 3. 

Based on the gartners TIME mapping, 4 applications fell into the “Eliminate” category, and the rest 
fell into the “Invest” category. Of these 4 applications, only | was an application with a strategic function, 
while the others were support functions. The BMN Plus application is an application that is considered to 
have a strategic function, but after mapping gartner’s TIME, this application was recommended to be 
eliminated, this was because this application had not been optimized enough by the user, so its function was 
considered insignificant to the organization’s success; the expected strategic functions were not supported by 
good IT quality. 

Another elimination recommendation falls on applications with a support function: e-KI, Kerjasama 
Online (Online Collaboration), and e-Arsip (e-archive). Aside from the fact that these applications do not 
have substantial business support for the organization, they are also built with poor IT quality. To confirm 
these results, we consulted experts. When this research was conducted, there was a process of significant 
changes to research institutions in Indonesia. The merging of research institutions in Indonesia into one 
organization called BRIN resulted in a difference in the entire management process. Therefore, even though 
the recommendations are considered appropriate, the results of the recommendations cannot be implemented. 
However, the output of the APM can still be used for decision-makers to make a more objective comparison 
of which applications can still be used on a larger scale. So, organizations can save costs more than building 
new applications even though they have functioned that old applications can fulfill. 
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Eliminate Tolerate | Invest Migrate 
e-Kl, Kerjasama SIDUPER, Bels, e- 
Online (Online Salary, SIEVITA, 
Collabaration), e- SIGAGAS, Sijempol, 
Support Arsip (e-Archive) SISKA, Spirit 
Fabiola, SIM Cepat 
Tanggap (Quick 
Key Operational Respon) 
High Potential Sispekin 
Strategic BMN Plus 


Figure 2. The result of combining McFarlan and gartner’s model 


5. CONCLUSION 

The staggering fact that Indonesia’s government registered as many as 400,000 digital applications 
raises our concern about how Indonesia’s government institutions manage their application portfolio. 
Surprisingly, no research has been done to expose how government institutes manage their existing and 
future applications. As a case study, this study looked at how BPPT, a government research institute, 
manages its application portfolio. Our findings are important in different aspects. First, we found that most of 
the digital applications registered on BPPT fall into the operational/management support category. This is 
contradictory, as BPPT is a research institute, where research is the main business value of an institution. 
Does this pattern apply to most institutions in Indonesia, where applications are seen as a management- 
supporting tool and not for core business support? Future research should use the different institutions as a 
research subject and compare the results with our findings. Second, we tested and confirmed the robustness 
of McFarlan’s strategic grid and gartner’s TIME model as a framework for the APM process. Our work will 
be among the first to apply the two frameworks in Indonesia’s context. IT decision-makers can utilize our 
findings to create a strategy that fits BPPT’s business value. However, in October 2021, through Presidential 
Decree No. 78, about the National Research and Innovation Agency, BPPT was officially merged into one 
new agency. Nevertheless, our findings can help align existing applications that BPPT has with the new 
agency's planned applications. APM can be at its best when all parties collaborate. The lack of application 
information will slow down the process, affecting the recommendation quality later. We strongly recommend 
conducting APM on the new institution by collaborating. 
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